Are User Reviews Systematically Manipulated? Evidence from the Helpfulness Ratings

There is a lot of evidence that people place great weight on online user reviews. And yet there are many reports of mischief in reviews, such as the January 2009 incident with a Belkin manager publicly offering sixty-five cents for positive reviews of their products. Is the high level of trust warranted when there are so many motives and opportunities for manipulating reviews? That question motivates this work. We propose a way to quantify the extent of manipulation in online user reviews. Our approach is based on the idea that the combination of the review data and the helpfulness ratings of the reviews that many sites now offer (the answers to the “was this review helpful to you?” question, what we call “metareview data”) provides clues to manipulative behavior. If a highly motivated segment of the reviewing population has an agenda to push for a product, then they would use both the reviews to express their views and the helpfulness ratings to reinforce those views: someone promoting a product would write a favorable review, mark other favorable reviews as helpful, and mark unfavorable reviews as unhelpful. That double voting leaves an identifiable trail in the data. We argue that such trails are circumstantial evidence of manipulation, and therefore call into question the credibility of the review data itself. Our findings are based on five data sets: elliptical trainers from Amazon (76 products, with 1,378 reviews in total, and 9,548 helpfulness votes), computer accessories from Amazon (370 products, 3,599 reviews, 16,495 votes), snack products from Amazon (1,201 products, 7,319 reviews, 11,742 votes), iPhone applications from the iTunes store (100 products, 7,509 reviews, 14,842 votes), and companies from Vanno, a company reputation website (586 companies, 7,636 submissions, 281,047 votes). In all of our data sets, we find that manipulation clearly is not the dominant mode of behavior. However, in the Amazon and iTunes data sets, we see that while manipulation is not dominant, it is still prevalent. Using our “minimum distance” approach, the models that predict manipulation are the best match for as low as 20% (for the elliptical trainers) and as high as 47% (for the snack products). The computer accessories and iTunes apps are in between. The Vanno data set shows almost no evidence of manipulation, at less than 1%.